7 research outputs found
Programmable Spectrometry -- Per-pixel Classification of Materials using Learned Spectral Filters
Many materials have distinct spectral profiles. This facilitates estimation
of the material composition of a scene at each pixel by first acquiring its
hyperspectral image, and subsequently filtering it using a bank of spectral
profiles. This process is inherently wasteful since only a set of linear
projections of the acquired measurements contribute to the classification task.
We propose a novel programmable camera that is capable of producing images of a
scene with an arbitrary spectral filter. We use this camera to optically
implement the spectral filtering of the scene's hyperspectral image with the
bank of spectral profiles needed to perform per-pixel material classification.
This provides gains both in terms of acquisition speed --- since only the
relevant measurements are acquired --- and in signal-to-noise ratio --- since
we invariably avoid narrowband filters that are light inefficient. Given
training data, we use a range of classical and modern techniques including SVMs
and neural networks to identify the bank of spectral profiles that facilitate
material classification. We verify the method in simulations on standard
datasets as well as real data using a lab prototype of the camera
Foveated Thermal Computational Imaging in the Wild Using All-Silicon Meta-Optics
Foveated imaging provides a better tradeoff between situational awareness
(field of view) and resolution and is critical in long-wavelength infrared
regimes because of the size, weight, power, and cost of thermal sensors. We
demonstrate computational foveated imaging by exploiting the ability of a
meta-optical frontend to discriminate between different polarization states and
a computational backend to reconstruct the captured image/video. The frontend
is a three-element optic: the first element which we call the "foveal" element
is a metalens that focuses s-polarized light at a distance of without
affecting the p-polarized light; the second element which we call the
"perifoveal" element is another metalens that focuses p-polarized light at a
distance of without affecting the s-polarized light. The third element is
a freely rotating polarizer that dynamically changes the mixing ratios between
the two polarization states. Both the foveal element (focal length = 150mm;
diameter = 75mm), and the perifoveal element (focal length = 25mm; diameter =
25mm) were fabricated as polarization-sensitive, all-silicon, meta surfaces
resulting in a large-aperture, 1:6 foveal expansion, thermal imaging
capability. A computational backend then utilizes a deep image prior to
separate the resultant multiplexed image or video into a foveated image
consisting of a high-resolution center and a lower-resolution large field of
view context. We build a first-of-its-kind prototype system and demonstrate 12
frames per second real-time, thermal, foveated image, and video capture in the
wild
Broadband Thermal Imaging using Meta-Optics
Subwavelength diffractive optics known as meta-optics have demonstrated the
potential to significantly miniaturize imaging systems. However, despite
impressive demonstrations, most meta-optical imaging systems suffer from strong
chromatic aberrations, limiting their utilities. Here, we employ inverse-design
to create broadband meta-optics operating in the long-wave infrared (LWIR)
regime (8 - 12 m). Via a deep-learning assisted multi-scale differentiable
framework that links meta-atoms to the phase, we maximize the
wavelength-averaged volume under the modulation transfer function (MTF) of the
meta-optics. Our design framework merges local phase-engineering via meta-atoms
and global engineering of the scatterer within a single pipeline. We
corroborate our design by fabricating and experimentally characterizing
all-silicon LWIR meta-optics. Our engineered meta-optic is complemented by a
simple computational backend that dramatically improves the quality of the
captured image. We experimentally demonstrate a six-fold improvement of the
wavelength-averaged Strehl ratio over the traditional hyperboloid metalens for
broadband imaging.Comment: 28 pages, 12 figure
MINER: Multiscale Implicit Neural Representations
We introduce a new neural signal model designed for efficient high-resolution
representation of large-scale signals. The key innovation in our multiscale
implicit neural representation (MINER) is an internal representation via a
Laplacian pyramid, which provides a sparse multiscale decomposition of the
signal that captures orthogonal parts of the signal across scales. We leverage
the advantages of the Laplacian pyramid by representing small disjoint patches
of the pyramid at each scale with a small MLP. This enables the capacity of the
network to adaptively increase from coarse to fine scales, and only represent
parts of the signal with strong signal energy. The parameters of each MLP are
optimized from coarse-to-fine scale which results in faster approximations at
coarser scales, thereby ultimately an extremely fast training process. We apply
MINER to a range of large-scale signal representation tasks, including
gigapixel images and very large point clouds, and demonstrate that it requires
fewer than 25% of the parameters, 33% of the memory footprint, and 10% of the
computation time of competing techniques such as ACORN to reach the same
representation accuracy.Comment: 14 pages, accepted to ECCV 202